Direct Zero-Norm Minimization for Neural Network Pruning and Training
نویسندگان
چکیده
Designing a feed-forward neural network with optimal topology in terms of complexity (hidden layer nodes and connections between nodes) and training performance has been a matter of considerable concern since the very beginning of neural networks research. Typically, this issue is dealt with by pruning a fully interconnected network with “many” nodes in the hidden layers, eliminating “superfluous” connections and nodes. However the problem has not been solved yet and it seems to be even more relevant today in the context of deep learning networks. In this paper we present a method of direct zero-norm minimization for pruning while training a Multi Layer Perceptron. The method employs a cooperative scheme using two swarms of particles and its purpose is to minimize an aggregate function corresponding to the total risk functional. Our discussion highlights relevant computational and methodological issues of the approach that are not apparent and well defined in the literature.
منابع مشابه
An efficient one-layer recurrent neural network for solving a class of nonsmooth optimization problems
Constrained optimization problems have a wide range of applications in science, economics, and engineering. In this paper, a neural network model is proposed to solve a class of nonsmooth constrained optimization problems with a nonsmooth convex objective function subject to nonlinear inequality and affine equality constraints. It is a one-layer non-penalty recurrent neural network based on the...
متن کاملAdaptive Predictive Controllers Using a Growing and Pruning RBF Neural Network
An adaptive version of growing and pruning RBF neural network has been used to predict the system output and implement Linear Model-Based Predictive Controller (LMPC) and Non-linear Model-based Predictive Controller (NMPC) strategies. A radial-basis neural network with growing and pruning capabilities is introduced to carry out on-line model identification.An Unscented Kal...
متن کاملSurface Tension Prediction of Hydrocarbon Mixtures Using Artificial Neural Network
In this study, artificial neural network was used to predict the surface tension of 20 hydrocarbon mixtures. Experimental data was divided into two parts (70% for training and 30% for testing). Optimal configuration of the network was obtained with minimization of prediction error on testing data. The accuracy of our proposed model was compared with four well-known empirical equations. The arti...
متن کاملLearning Sparse Neural Networks through L0 Regularization
We propose a practical method for L0 norm regularization for neural networks: pruning the network during training by encouraging weights to become exactly zero. Such regularization is interesting since (1) it can greatly speed up training and inference, and (2) it can improve generalization. AIC and BIC, well-known model selection criteria, are special cases of L0 regularization. However, since...
متن کاملRethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers
Model pruning has become a useful technique that improves the computational efficiency of deep learning, making it possible to deploy solutions in resourcelimited scenarios. A widely-used practice in relevant work assumes that a smallernorm parameter or feature plays a less informative role at the inference time. In this paper, we propose a channel pruning technique for accelerating the computa...
متن کامل